@slevis-lmwg I did some work on this. And I'm viewing it as at a place for you to complete the tag for. Let me know if there's anything we need to go over. Adding more testing before we go ahead and make several tags seems like the best ordering of things.

I also went through the MOSART issues and flagged a couple as things we perhaps should work on. As well as marking a bunch as lower priority that we don't need to worry about right now.

Submitting the mosart test-suites next.

Sorry, I have to partly back out 891a45d because now i'm getting ERROR: Invalid compset name, I2000Clm60BgcCru

Thinking about this I think we probably should use mostly the same compsets for these MOSART tests. Answers will be different for Sp vs. BgcCru cases -- but in terms of MOSART do we need to test with both SP and BGC? And with different forcing? We should exercise all the MOSART options -- but maybe use the same general compset for all. The differences that do matter are 1850 vs 2000 vs HIST as well as when CISM is used.

Part of the philosophy of spanning many compsets is to show that MOSART continues to work under standard things that change answers. But, I don't think we've seen tests show problems under different compsets that primarily change CLM answers. So I think we should maybe drop this and simplify the testing compsets.

@slevis-lmwg should we maybe have a quick meeting about this?

On izumi this fails, including with vanilla mosart
./run_sys_tests -s mosart -c mosart1.1.02-ctsm5.2.029 -g mosart1.1.03-ctsm5.3.009

Traceback (most recent call last):
  File "./run_sys_tests", line 20, in <module>
    main(cime_path=cime_path)
  File "/fs/cgd/data0/slevis/git/sep_inst_flds_iss1059/python/ctsm/run_sys_tests.py", line 93, in main
    extra_create_test_args=args.extra_create_test_args,
  File "/fs/cgd/data0/slevis/git/sep_inst_flds_iss1059/python/ctsm/run_sys_tests.py", line 235, in run_sys_tests
    running_ctsm_py_tests=running_ctsm_py_tests,
  File "/fs/cgd/data0/slevis/git/sep_inst_flds_iss1059/python/ctsm/run_sys_tests.py", line 689, in _run_test_suite
    suite_compilers = _get_compilers_for_suite(suite_name, machine.name, running_ctsm_py_tests)
  File "/fs/cgd/data0/slevis/git/sep_inst_flds_iss1059/python/ctsm/run_sys_tests.py", line 778, in _get_compilers_for_suite
    test_data = get_tests_from_xml(xml_machine=machine_name, xml_category=suite_name)
  File "/fs/cgd/data0/slevis/git/sep_inst_flds_iss1059/cime/CIME/test_utils.py", line 43, in get_tests_from_xml
    for comp in comps:
TypeError: 'NoneType' object is not iterable

@ekluzek @jedwards4b let me know if we can live without testing on izumi or if we need this fixed before I proceed.

@slevis-lmwg we should figure out what is up with testing on Izumi, and get it resolved and working. This problem isn't something specific for MOSART it must be something about the CTSM version being used here...

@jedwards4b don't worry about this -- we'll figure it out...

@mvdebolskiy @mvertens
From meeting with @ekluzek
we are updating the mosart testlist. See one of the next commits that I will push soon.

@jedwards4b can you approve this now? It looks like it's held up until you approve it. We could dismiss your review -- because it was addressed. But, better for you to just click approve.

Thanks.

Generating baselines:
./run_sys_tests -s mosart -c mosart1.1.02-ctsm5.2.029 -g mosart1.1.02-ctsm5.3.009
OK izumi
OK derecho

Comparing to baselines:
./run_sys_tests -s mosart -g mosart1.1.03-ctsm5.3.009 -c mosart1.1.02-ctsm5.3.009
OK izumi
IN PROG derecho: /glade/work/slevis/git/hXi_hXa_tapes_iss1059/tests_1106-135317de

@slevis-lmwg @mvertens
The testlist looks ok to me for now, If we need to adjust it for our machines, you'll just get a PR here from NorESMhub fork.

• When mosart test suite completes, I will make the tag and merge

New tag
mosart1.1.03